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Abstract 


This paper presents the design, implementation, and evaluation of an interactive hypermedia 
math program which focuses on multiplication and problem solving. The program contains a 
thematically meaningful story about a lost dog named Addy. The designers of the program based 
their work on instructional design principles defined by Walter Dick and Lou Carey (1990) in 
their book The Systematic Design of Instruction. Upon completion, the designers implemented 
and evaluated the math program in a combined second and third grade classroom. The evaluation 
of the program provides support for the integration of computer assisted instruction into the 


elementary math curriculum. 
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CHAPTER ONE: INTRODUCTION 


In their 1990 survey of reputed computer-using teachers, Karen Sheingold and Martha 


Hadley found that some teachers at all grade levels use computer software to enhance the 
curriculum with intellectually exciting educational experiences. These teachers abandoned the 
traditional view of computers as substitutes for paper and pencil worksheets and as rewards for 
those that complete their classwork. Instead, these teachers employed a wide variety of 


computer software including simulations, graphing programs, writing tools, communications 


software, and problem solving programs to address curricular goals. 

While there are numerous software programs available to educators, these programs do 
not meet every classroom’s needs. Trying to instruct a class of students using a computer 
program that is either too difficult or not engaging enough is much like trying to dress a small 
child in clothes that are too small or too large. Ultimately, despite persistent effort, the child is no 
better off and very uncomfortable. This was the situation in a combined second and third grade 
Classroom in Albemarle County, Virginia. The classroom teacher was unsuccessful in her search 
for math software that reinforces multiplication concepts. The software she found had too much 
text for the students to read and did not engage the students. 

In order to help the teacher with this problem, we decided to apply our knowledge of 
instructional technology and to create software that would fit this classroom's needs. Our mission 
was to develop a fun, educational, and user friendly math program to reinforce multiplication in 
the context of problem solving. We wanted to present the concepts in this context because of the 


current emphasis on incorporating problem solving into all areas of math education. 
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To accomplish our mission, we researched and applied a variety of principles for 
developing instruction. Chapter Two will present our findings on designing instruction in general 


and will discuss specific applications of these ideas to software design. Chapter Three will follow 


with a discussion of how we applied these principles to our program. In Chapter Four, we will 


offer the evaluation of our program. We conc!ude with Chapter Five in which we will summarize 


the study and discuss implications for teaching and future research. 
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CHAPTER TWO: REVIEW OF INSTRUCTIONAL DESIGN LITERATURE 


Research indicates that important design considerations for authoring a hypermedia 


program are similar to those considerations taken when writing a lesson plan. Good instructional 


design for lessons long preceded computers in the classroom. To arrive at current beliefs about 
design, “we have taken what is known about designing good instruction and added to it what we 
have learned about the power of the visual” (Dana and Handler, 1995, p.252). The following 
paragraphs contain currently accepted design considerations for creating effective instruction. 
The steps toward developing effective instruction are: identify the instructional goal, identify the 
audience, determine the instructional strategy, deveiop instructional materials, and conduct a 
formative evaluation. These principles are adapted from and are based primarily upon Walter 
Dick and Lou Carey’s (1990) work, The Systematic Design of Instruction. Specifically, we 
discuss incorporating these priuciples into software design. 
Identify the Instructional Goal 

Instruction is the solution to a problem. Design begins, therefore, with the determination 
of this problem and the subsequent determination of the instructional goal. Problems should be 
identified through what Dick and Carey (1990) call a needs assessment. This assessment 
identifies a gap between what is and what should be. 

Identify the Audience 

Once the instructional goal has been identified, a designer must turn his or her attention to 
the audience or the “target population” that he or she is aiming to “hit” with instruction. The 
entry behaviors and characteristics of the audience is essential information. “In order to have 


effective instructional materials or, for that matter, any type of successful instructional 
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experience, there must be a match between students and instruction” (Dick and Carey, 1990, 
p.83). 


In any group that is receiving instruction, there will be variation. Learner characteristics 


are, therefore, described in terms of the average student. Necessary information about the target 
group includes age, grade level, and knowledge. Also relevant to instruction are the interests of 
the group. Dick and Carey caution designers, however, about making generalizations in terms of 
these learner characteristics. 

Too often designers make references about the characteristics of the leamers 


without actually verifying them. Based upon stereotypes of the learners it might be | 


assumed that fourth grade boys would be interested in baseball...Quite often these 

assumptions reflect the interests of the designer ...and are not at all reflective of the target 

population. Therefore it is important to observe and interview members of the target 
population to determine not only their status, vis-a-vis the entry behaviors, but their 
general ability level, previous experiences, and expectations about instruction (Dick and 

Carey, 1990, p. 89). 

In addition to learner characteristics, the designer must determine the entry behaviors or 
skills that must be mastered before beginning instruction. These entry behaviors are the initial 
blocks upon which the instruction will be built. Entry behaviors are not simply a list of all things 
that the students know or can do. They should include only those skills related to the topic and, 
consequently, necessary to begin the instruction. At this stage, the designer can err in one of two 
directions. First, if the instruction is designed for only the brightest students in a population, the 


majority of the students will not possess the entry behaviors needed to benefit from the 
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instruction. When this happens, the instruction will be ineffective because the students will 


become frustrated. On the other hand, the designer could assume that the target audience has few 
or no skills when in actuality they already possess much of the knowledge covered in the 
instruction. This poses a problem because time is wasted as learners study cbjectives they have 
already mastered. Thorough knowledge of the learners will alleviate the chance of these errors. 
Determine the Instructional Strategy 

After determining the instructional goal and describing the target population, the designer 
must generate an instructional strategy. The designer knows where he or she is going, who he or 
she is taking, and is now left to plan the course. “An instructional strategy describes the general 
components of a set of instructional materials and the procedures that will be used with those 
materials to elicit particular learning outcomes from the students” (Dick and Carey, 1990, p.162). 
The five major components of an instructional strategy include: preinstructional activities, 
information presentation, student participation, testing, and follow through. 

Preinstructional Activities 

Prior to beginning instruction, there are three factors that a designer should consider. The 
first factor is motivating the students. The second factor is informing the students of what they 
will learn. The third factor is ensuring that the learners have the prerequisite knowledge and skills 
needed to begin instruction. 

The level of student motivation is a key determinant in the design process. Research gives 
particular emphasis to engaging students with problem-based learning. This model includes 

motivating the learner by demonstrating the practical applications and importance of the 


knowledge, providing a conceptual description of the skill, demonstrating the application 
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of the knowledge to practical problems, providing practice opportunities with support 

in the form of scaffolding, and facilitating transfer through guided reflection on the activity 

to integrate the practical issues with the underlying conception (Quinn, 1997, p. 2). 
While instruction may be able to gain a learner’s attention for a short period of time, it is difficult 
to sustain this interest if the learner does not perceive that the material is relevant to him or her 
and that the material has practical implications for his or her life. 

Considerable research has also been devoted to a discussion of multimedia’s ability to 
engage and motivate learners. 

Engagement serves as the “bridge” to learners and is built best by ensuring that 

students can relate to the information (text and graphics) presented. The relationship of 

presented material to a learners existing knowledge base, each known as a link, is a critical 

success factor in using multimedia for instruction (McFarland, 1995, p.67). 

In general, a person will learn better and be more motivated if many links to prior experiences are 
included in the current material. 

Another concem prior to beginning instruction is informing the students of the objectives 
for instruction. Designers must tell the learners what is expected of them so that they can focus 
their study strategies and efforts on these outcomes. 

The last factor to consider before starting instruction is ensuring that the learners have the 
prerequisite knowledge and skills needed for effective instruction. The purpose of this activity is 
to make certain that instructional time will not be wasted and that the learners have the means to 


accomplish the objectives. 
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Presentation of Information 


Once the preinstructional activities have been completed, the designer is ready to consider 
the presentation or sequencing of the information. One appropriate method for sequencing is to 
begin with the lower level tasks and then progress to more difficult tasks. Another appropriate 
method is to construct a non-linear environment using hypermedia. This medium provides an 
exploratory learning system in which the learner controls the delivery of material. Stanton and 
Stammers (1990) suggest that the reasons why a nonlinear environment might be superior are that 
it (a) allows for different levels of prior knowledge, (b) encourages exploration, (c) enables 
subjects to see a subtask as part of the whole task, and (d) allows subjects to adapt material to 
their own learning style. 

Also in terms of presentation, the designer must determine exactly what information, 
concepts, rules, and principles need to be presented to the learner. This implies a focus on the 
types and number of examples that will enhance the concept being presented. 

Student Participation 

The next major component in the instructional strategy is student participation and 
feedback. A powerful aspect of the learning process is practice with feedback. Feedback simply 
gives learners information about their performance. Shneiderman (1983) and Hutchins, Hollan & 
Norman (1986) suggest a tight coupling between action and feedback both in the form of the 
communication, and in the time between action and response. The nature of feedback is also 
important. “It should be balanced so that it focuses on both the successes and failures in the 
rehearsal. Focusing only on errors may cause learners to perceive that nothing they did was 


meritorious, which is seldom the case” (Dick and Carey, 1990, p.168). 


Development of a Math Program 11 


Testing 


At this point, the designer must decide what his or her strategy will be for testing. Tests 
are a valuable communication device that lets the learners know how they are progressing through 
the material. Issues such as when during instruction the test will be administered, what skills it 
will cover, and the what form it will take are important considerations for the designer. 

Follow Through 

Part of good instruction is developing follow-up activities for students to complete after 
instruction. Remediation may be necessary for those who struggle through the instruction and 
enrichment could be offered to the students who master the material with ease. 

Develop Instructional Materials 

The next step, after developing the instructional strategy, is to assemble the materials that 
will be used during instruction. A key suggestion at this stage is “don’t reinvent the wheel.” If 
someone has already developed materials that are appropriate for the delivery of instruction, the 
designer should use them. If no appropriate :naterials exist, the designer should begin the 
development process. 

“One of the most interesting and challenging decisions in the instructional design process 
is the selection of the medium or media that will be used to deliver the instruction” (Dick and 
Carey, 1990, p.202). This decision depends on the designer’s knowledge of what will be taught, 
how it will be taught, how it will be tested, and who the learners will be. Weiser and Gagne 
(1983) have published a flow chart that details how to choose the appropriate medium for 
instruction. While using the flow chart, the designer answers several questions that lead him or 


her to several media suggestions. Important factors for the designer to keep in mind when 
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choosing among these suggestions are projected availability of media and the cost effectiveness of 
the media. 

Research on the best media to teach intellectual skills suggests that students should be 
provided precise corrective feedback to responses made during the instruction process. Dick and 
Carey (1990) agree that in order to provide responsive feedback, there is a need for such 
interactive media as computer based instruction. 

Conduct a Formative Evaluation 

Formative evaluation is the process used by instructors to gather data that will help them 
revise their instruction and make it more effective. Dick and Carey (1990) suggest a three phase 
evaluation process. The first phase is a one-to-one or clinical evaluation. The purpose of this 
phase is to remove the obvious errors and obtain initial reactions from a learner. The second 
phase of formative evaluation is the small group evaluation. The members of the small group 
evaluation team should be as representative as possible of the target population since this step 
gathers information about any remaining learning problems that actual students may encounter. 
The purpose of the final phase, the field trial, is to determine if the instruction can be used in the 
environment for which it was designed. For this reason, the designer must attempt to find a field 


situation that closely resembles the intended users’ situation. 
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CHAPTER THREE: DEVELOPMENT OF OUR HYPERMEDIA MATH PROGRAM 
Identification of Our Instructional Goal 
The development of our hypermedia math program began with the identification of a 

problem and the subsequent determination of an instructional goal. A teacher at an elementary 
school in Albemarle County, VA was unsatisfied with the current math software programs 
available to her, primarily because the design of these programs did not match the needs of her 
students. As indicated by the teacher, her students had a tendency to lose interest in the 
programs because of large amounts of text, few opportunities for interaction, and inappropriate 


levels of challenge. 


After extensive discussions with the teacher and with each other, we decided to help the 


teacher solve this problem by developing a math program that would meet the needs of her 


students. Our instructional goal was to create a program that would intrinsically engage the 


students and would lead them through an interactive experience that would enhance their ability 
to solve mathematical problems. 
Identification of Our Audience 

Once we identified the problem as well as our instructional goal, we focused our attention 
on the students for whom we were designing the program. Our first task was to identify the 
students’ general characteristics and interests. Our second was to identify specific entry 
competencies such as knowledge, skills, and attitudes about our topic. 

General Characteristics and Interest 
The students involved with our math program were members of a second and third grade 


combination class and ranged in age from seven to nine years old. The class consisted of thirteen 
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girls and nine boys, all of similar ethnicity and socio-economic status. The ability levels of the 
students spanned from low ability (special needs) to high ability (gifted). 

Through informal discussions and written autobiographies, the students indicated that 
they all shared a common interest in pets. The majority of the students had at least one pet at 
home and some had as many as three or four. They also indicated that they enjoyed traveling and 
visiting ditierent locations. 

Entry Competencies 

At the onset of our field work, the students were in the process of learning multiplication 
facts for numbers 0-9. The teacher’s goal was to move from this level of multiplication to two- 
digit multiplication by the projected completion of our math program. In addition to learning 
multiplication, the students were also learning to use the information presented in word problems 
to write and solve mathematical equations. 

In regards to technology, the students demonstrated a solid knowledge base as well as 
advanced user skills. They were familiar with and could independently use the classroom 
computer’s hardware as well as its software. Upon further investigation, we found that the 
students had an abundance of technology experience both in and out of the classroom. The 
teacher consistently and effectively integrated technology into her units of study. In addition, the 
majority of the students used computers in their homes for daily activities. 

Informal discussions with the students and with the teacher revealed that the students 
held positive attitudes toward math as well as toward math software, as long as their experiences 
with these topics were “fun.” When instruction did not provide an element of fun, the students 


had a tendency to lose interest and to develop negative attitudes toward math. 
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Identification of Our Instructional Strategies 

After determining the instructional goal and describing our target population, we began to 
generate our instructional strategy. The following paragraphs are descriptions of the three general 
components that we used to create a math program that is engaging, interactive, educational, user- 
friendly, and fun. These components included preinstructional activities, presentation of 
information, and student participation. 

Preinstructional Activities 

Our preinstructional activities consisted of selecting strategies for motivating the students 
to use our program, for informing the students of their tasks, and for ensuring that the students 
had the prerequisite knowledge and skills needed to use the program. 

Motivating the Students 

Our first concern was motivating the students to use our program. In response to our 
research on student motivation, we decided to integrate into our program a thematically 
meaningful story line based on the students’ common interest in owning pets. Specifically, we 
designed our story line to center around a lost dog named Addy who must be located by his 
owner (see Figures | and 2). 

In addition to a thematically meaningful story line, we decided to integrate a problem- 
based learning model into our program to increase student motivation. The model we selected 
involved demonstrating the practical application and importance of the knowledge, providing a 
conceptual description of the skill, and demonstrating the application of the knowledge to 


practical problems. 
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In order to accomplish all of the tasks involved in the problem-based learning model, we 
decided to expand our story line by having the students look for Addy in different locations 
within a tc wn (see Figure 2). At each of the locations, we opted to present the students with a 
“real-life” problem that required them to reflect on multiplication as it is used outside the 


classroom and to use their skills at a conceptual level (see Figure 3). 


Informing the Students 


Our second concern was informing the students of our objectives for the math program. 


We wanted the students to know exactly what we expected of them so that they could focus 
their strategies and efforts on those outcomes. 

In order to specify our non-instructional objective, finding Addy, we choose to provide 
the students with a brief introduction and statement of purpose at the beginning of the program. 
The second screen contains a brief descriptions of the students’ task while using the program. 
We tell the students what we want them to accomplish and how to be successful (see Figure 2). 

When considering our instructional objective, enhancing the students’ multiplication 
skills, we felt as though a brief introduction at the beginning of the program would not give the 
learners the opportunity to clearly understand our purpose. For this reason, we chose to conduct 
an informal discussion prior to implementing the program. In this discussion, we wanted to 
clearly communicate our instructional purpose and to clarify any misconceptions held by the 
students. 

Prerequisite Knowledge and Skills 
Our third concern was ensuring that the students had the prerequisite knowledge and 


skills needed to complete our program. Through informal interviews and observations, we found 
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that both the students’ knowledge base and their skills were sufficient to accomplish our non- 
instructional as well as our instructional objectives. 
Presentation of Information 

Once we completed our preinstructional activities, we began to develop the second 
component of our instructional strategy, the presentation of our information. This component 
involved sequencing the information to be presented as well as selecting design principles and 
guidelines for effective presentation. 
Sequencing the Information 

We felt that a non-linear environment would be the best method for sequencing our 
information because it would provide the students with an exploratory learning system. They 
could explore the program at their own discretion and could control the delivery of material by 
choosing where they wanted to look for Addy. By making the students the locus-of-control and 
granting them a perception of choice, we believed that our efforts to instruct, engage, and interact 
with the students would be more effective. 
Selecting Principles and Guidelines 

In their book Instructional Media and Technologies for Learning, Heinich, Molenda, 
Russell, and Smaldino (1996) point out that some designers have a tendency to overemphasize 
text and to ignore the benefits of visual imagery. As a result, they create instructional 
environments that are neither engaging nor educational. To avoid this type of failure, we chose to 
follow certain principles and guidelines related to the effective use of visual imagery. The 
majority of these principles and guidelines were taken from the third chapter of Instructional 


Media and Technologies for Learning. 
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For the purpose of information and instruction, we focused our attention on three areas of 
visual design as suggested by Heinich et al. (1996): ensuring legibility of text and graphics, 
reducing the effort required to interpret the information presented, and increasing the students’ 
active engagement with the information. | 

ensuring legibility of words and images. 

A good visual cannot even begin to do its job unless all users can see ‘he words. Our goal 
was to remove as many obstacles as possible that might impede the engagement and instruction 
of the students. To accomplish this goal, we employed design principles related to lettering style, 
size, and color. 

When choosing the style of lettering, we elected to use a plain style that was consistent 
and harmonious with other visual designs. In addition, we used no more than two different 
lettering styles per screen (see Figure 1). 

To further increase legibility, we used lowercase letters, adding capitals only where 
required. In HyperCard Stack Design Guidelines released by Apple Computer, Inc. (1989), the 
authors claim that it is much easier for people to read a mix of uppercase and lowercase letters 
than it is for them to read only uppercase. 

Because lowercase letters are obviously smaller than uppercase letters, we knew that the 
size of our lettering would be crucial to the legibility of our screens. Following a guideline 
suggested by Heinich et al. (1996), we made our lowercase letters 1/2 inch high for each ten feet 
of viewer distance. 

After making decisions about lettering style and size, we began to select the color of our 


lettering. We wanted the color to contrast with the background color both for the sake of simple 
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legibility and for the sake of emphasis. With this in mind, we decided to make the majority of our 
lettering black. We only used a different color when we wanted to draw the users attention to a 
specific word. 

reducing effort. 

As designers, we wanted to convey our information in such a way that the students 
would expend little effort making sense out of what they saw. The idea was to establish an 
underlying pattern that would decide how our user’s eyes would flow across our screen. 

Our first step was to position the primary elements within the screen so that they had a 
clear visual relationship to each other. We established this visual relationship by aligning the 
edges of our elements on the same imaginary horizontal or vertical line which was parallel to the 
edges of the screen. 

Our second step was to put the visual elements into a shape that would be familiar to the 
viewer. Our goal was to use a pattern that would attract and focus the user’s attention as 
effortlessly as possible. To accomplish this goal, we placed text at the top of the screen and 
graphics at the bottom whenever possible (see Figure 3). 

Color can also make a difference in the effort required to interpret information. When 
designing our program, we followed two of McFarland’s (1995) guidelines for using color to 
reduce effort. First, we used soft, non-intrusive colors such as pastels or soft grays for screen 
backgrounds. Eyes have a tendency to become fatigued when exposed to highly saturated colors 
for an extended period of time. Second, we used a consistent color scheme throughout the entire 
program. When a different color scheme was necessary for emphasis, we made sure that we 


selected colors which were complementary to our original scheme. 
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Without consistency, we knew that the guidelines related to alignment, shape, and color 
would be useless. Heinich et al. (1996) claim that as viewers go through a series of images, they 
begin unconsciously to form a set of rules about where information will be located on the screen. 
The more often the arrangement conforms to these rules, the more the viewers trust the rules. 
Every time the arrangement breaks the rules, the viewer has to expend mental energy deciding 
whether this is a deliberate exception or whether the rules need to be revised. To avoid this 
unwanted expenditure of mental energy, we placed similar elements in similar locations, used the 
same text style, and used the same color scheme throughout our program. 


increasing active engagement. 


When considering active engagement, one of our goals was to make our design as 


appealing as possible. We wanted to get the students’ attention and to entice them into thinking 


about the information presented. The following is a summary of our attempts to capture and hold 


our users’ attention and to actively engage them in the learning process. 

Because users pay attention as long as they are getting novel stimuli or new information, 
we decided to incorporate as many different situations and problems as possible. Within four of 
the six locations in the town, we give the users at least two new and unrelated situations and 
problems. For example, while looking in the park, the users have the option of visiting the 
climbing tree to count apples, the bike trail to calculate traveling distance, or the picnic area to 
figure out how much food a family is missing (see Figures 4,5,6, and 7). 

To further increase opportunities for active engagement, we provided users with the 
feeling of directly manipulating our program. We allowed them to respond to our visual display 


by manipulating objects and representations on the screen. In addition, we attempted to promote 
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active engagement by stimulating the curiosity of the user. Our most effective element for 
stimulating curiosity was the non-deterministic outcome of the program. The users never know 
where Addy is hiding. 


Student Participation 


In order for students to completely participate in the learning process, instructional 


methods and materials must provide opportunities for practice with feedback. Research indicates 
that a tight coupling between action and feedback is important both in the form of 


communication, and in the time between action and feedback. In addition, the feedback must be 


directly related to the action and must be balanced so that it focuses on both the students’ 
successes and failures. 

Applying these ideas to our program involved developing methods to provide rapid, 
relevant, and balanced feedback. Our first method was to use quick and non-intrusive transitions 
between screens to immediately inform the students of their success or failure in solving the 
selected math problem. We felt that this would eliminate the possibility of confusion and would 
keep the problem in the students’ minds. 

Our second method was to embed the feedback into the context in which the students 
performed the action. We wanted the feedback to be relevant and related to the problems that we 
asked the students to solve. To accomplish this goal, we decided to rely on visual images to relate 
actions and feedback. For example, if a problem involved money, then the student would have to 
click on money to receive feedback and to return to the problem (see Figures 8 and 9). 

Our third method was to design separate yet related screens for different types of 


feedback. If the user selected an incorrect answer, we wanted them to arrive at a screen that 
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would encourage them to try again and to select the correct answer. We accomplished this by 
placing the statement “Oops! Not quite right. Try Again” on a dull and visually unattractive 
screen (see Figure 9). If the user selected the correct answer, we wanted them to arrive at a screen 
that gave them the perception that their efforts were meritorious. We accomplish<d this goal by 
placing the statement “Way to go! You are correct” on a visually exciting screen (see Figure 10). 
Determination of Our Instructional Materials 

The next step after developing our instructional strategy was to assemble the materials 
that we would need to complete our math program. These materials consisted of a storyboard, 
clip art, and a hypermedia program. 

Storyboard 

Before we sat down at the computer to actually design our math program, we spent a 
great deal of time designing a storyboard to help organize the sequence of our computer screens 
as well as the layout of each screen. We made simple sketches of the visuals that we planned to 
use and put them on individual index cards. We also included narration and production notes on 
these cards. Once we developed all of the cards, we color coded them according to their topic and 
mounted them in rough sequence on poster board. 

Clip Art 

After completing the storyboard, we began to search for images that we could use to 
represent our sketches. We felt that clip art images would be more effective than hand-drawn 
images because they are more attractive to the eye and because they would save us valuable time 
when creating the screens. With over 185,000 clip art images at our side, we found a match for 


every visual that we sketched on our storyboard. Unfortunately, some of the images we selected 
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were not sensitive to biases or stereotypes related to gender, ethnicity, ability, etc. Instead of 


discarding these images, we chose to edit them by changing those aspects that supported these 


biases and stereotypes. 
Hypermedia Program 
We based our decision to use HyperStudio on our familiarity with the program as well as 
on suggestions made by several technology instructors. We felt that HyperStudio would allow us 
to accomplish our instructional objectives and to employ our instructional strategies. 
As our program approached completion, we discovered that we could not effectively hide 
and show different cards as needed. Our knowledge of HyperLogo, the scripting language for 


HyperStudio, was insufficient for us to accomplish this task. 


Following the suggestion of a technology instructor, we transferred all of our work into a 
hypermedia program known as Oracle Media Objects (OMO). We found that the scripting 
language in OMO was much easier to use and that it would, in fact, allow us to hide and show the 
cards as needed. 

Our Evaluation Procedures 

The evaluation of our program occurred in three phases. The first phase involved a clinical 
evaluation in which we removed obvious errors and obtained initial reactions from a technology 
instructor. The second phase involved a field evaluation. While the targeted students used the 
program for the first time, we recorded informal observations of their experiences as well as their 
comments. The third phase involved a student evaluation survey in which we attempted to 


obtain information related content, motivation, and design (see Appendix). 
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Conclusion 


In this chapter, we presented an in depth discussion of how we applied our research on 


designing instruction to the math program we created. The next chapter will elaborate on the 


evaluations of our program and will analyze the results of these evaluations. 
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CHAPTER FOUR: EVALUATION AND RESULTS 
Introduction 

As stated in Chapter 3, the evaluation of our math program consisted of three stages: a 
clinical evaluation, a field evaluation, and student evaluations. The following paragraphs will 
elaborate on these evaluations and will summarize the results. 

Clinical Evaluation 

Our clinical evaluation involved removing obvious errors in the program. These errors 

consisted of spelling mistakes, grammatical mistakes, and aspects that did not adhere to the 


design principles and guidelines we selected. 


In addition to correcting errors, we also obtained initial reactions to our program from a 


technology instructor. As she used the program for the first time, she gave us valuable insight 
into one area of the program that might be confusing to the students. This area was navigation. 

In our program, we placed invisible buttons over clip art and linked these buttons to other 
screens. We assumed that the students would know to click on these images to navigate through 
the screens. Our technology instructor pointed out that this assumption may cause confusion. 
For this reason, we elected to add labels that would tell the students exactly what to do as well as 
where to go. 

Field Evaluation 

After completing the program and conducting a clinical evaluation, we presented the 
program to the students for whom it was designed. We took the students to a computer lab 
where we asked them to use the program. While the students were working, we recorded as many 


of their comments as possible. Examples of the students’ comments include: “I found Addy “X” 
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times”, “ Yes! We found Addy twice. This is an awesome game.”, “I’ve already seen this 
problem so I know the answer.” 

In addition to recording comments, we also recorded informal observations of the 
students’ experiences. We noticed that the students had difficulty when solving some of the math 
problems because they did not have a means of working them out by hand. Many of the students 
struggled to solve the problems in their heads instead of asking for paper or manipulatives. We 
also noticed that once the students solved a problem correctly, they had a tendency to remember 
the answer and automatically click on it upon returning to the screen. They did not waste any 
time reading or solving the problem again. 

The implications of these results are numerous. First, we need to provide the users of our 
program with a means of solving the multiplication problems by hand. One method would be to 
give the students paper and pencil or other manipulatives when using the program. Another 
would be to devise on screen notepads and manipulatives for the students to use. This method 
would be more effective in maintaining interaction between the student and the program. 

Second, we need to develop a method of changing the math problems every time a student 
restarts the program. The most effective way to accomplish this task is to create a variety of 
problems and then to script the program so that it randomly assigns one of these problems to the 
correct screen. 

Student Evaluations 
Once we implemented our program in the field setting, we asked the students to complete 


a survey to evaluate the program (see Appendix). This survey focused on three areas: content, 
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motivation, and design. It also included a section where students could make comments related to 
their experiences with the program. 

The results of the survey were overwhelmingly favorable. All of the students indicated 
that they understood the purpose of the game, could do the math problems, and could navigate 
through the program with ease. Eighty-three percent of the users liked the pictures as well as 
looking for Addy. Ninety-four percent of the students agreed that they would use the program 


again (see Tables 1,2, and 3 for details). 
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CHAPTER FIVE: SUMMARY AND CONCLUSIONS 


There are times when we as teachers will be faced with a concept that must be taught but 


for which no appropriate instructional material exists. In this situation, we believe that if it is not 


out there, then create it. The time spent in creation of this material will be duly repaid in less time 
spent reteaching a topic as a result of ineffective activities. Some teachers will elect to create 
instructional materials on their own. Others will choose to make the development of their 
materials a classroom project. 

For those teachers who wish to design their own software, the guidelines presented in our 
paper will be extremely useful. Combining these guidelines with the learning characteristics and 
entry behaviors of their students will allow these teachers to create successful instructional 


software. 


More importantly, however, we see this project having ramifications for self-directed, 
cooperative learning. If students are taught these design principles, they could then be instructed 
to work in cooperative groups to design a location where Addy could hide. They would need to 
determine everything from graphics to the problem that the user must solve. They would also 
have to select incorrect answers that would be viable choices. “Some of the best thinking results 
when students try to represent what they know. Representing knowledge as a mindful task can 
be enabled by cognitive tools such as hypermedia construction software” (Jonassen & Reeves, 
1996, p.696). Going through this process would ingrain math concepts as well as empower the 
students to create their own program. 

Designing, implementing, and evaluating our program was a beneficial experience for us. 


We were afforded the opportunity to apply our knowledge of instructional technology and to 


Zu 
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gain confidence in our ability to create instructional material to meet a specific classroom’s needs. 
Throughout our careers, we will be faced with the dilemma of fitting a child into “pants” that are 
either too small or too large. Hopefully, as a result of this experience, we will choose to take the 


time to design “pants” that are a perfect fit. 
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Table 1 
Results of student evaluations 


3=Yes 2=Not Sure 1=No 
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Table 2 


Average score for each survey question 
Average Score for Each Question 
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Purpose 
Problems 
Navigation i a 
Pictures “3 
Looking 
Play Again 


Question 
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Table 3 


Average score of individual evaluators 


Average Score of Individual Evaluators 
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Figure Captions 
Figure 1. Thematically meaningful story line. 
Figure 2. Presentation of non-instructional objective. 
Figure 3. Using skills at a conceptual level. 
Figure 4. Different situations and problems. 
Figure 5. Different situations and problems. 
Figure 6. Different situations and problems. 
Figure 7. Different situations and problems. 
Figure 8. Related and relevant feedback. 
Figure 9. Related and relevant feedback. 


Figure 10. Related and relevant feedback. 
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The Adventures of Addy 


Created by. Wendy Edwerds and Amy Schumacher 


Meet Addy. She lives and plays in 
Sunny Brook. Most of the time, 
Addy stays at home but 
sometimes her curiosity gets the 
best of her. To see what she is up 
to now, click on her tail. 


Welcome to Sunny Brook! 
TNC While you are visiting, we 
’ 


See a 
v , Is missing. se 
E ho us find Fa, 


You can begin your search 
24 clicking on one of the 
six locations you see here. 
If Addy is not in the first 
place you look and you 
want to visit another place, 
click on the compass rose 
like the one below. Happy 
Hunting! yy 


36° BEST COPY AVAILABLE 
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in the deli you find Marva preparing a customer's order. She 
needs to know how much to charge for 3 pounds of ham at $3.00 
per pound and 1 pound of turkey at $4.00 per pound. How much 
will these ems cost together? 


Figure 3 


: te Welcome to Grove's Point Park! 


8 If you want to look for Addy, click Hass 
Bon the family, the tree, or the bikers 
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While collecting leaves for 
her science project, 
Aileen noticed that there 
were 12 branches on the 
climbing tree. She also 
noticed that each branch 
held 8 apples. How many ¢ 
apples are on the 
climbing tree? 


During their ride, Mr. and Mrs. Wiitsee stopped at 6 different rest 
areas. The distance between each rest area is 2 miles. How many 
miles did the Wiltsee’s travel? , 
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The Beckmans are having a picnic in the 
park. Unfortunately, their food is being 
carried away by 9 ants. If each ant takes 
3 pieces of food, how many pieces will 
the Beckmans lose? 


I 


Latasha, the teller, is counting the 
money in Pete's piggy bank. There are 
11 $5 dollar bills and 3 $10 dollar bills 
in the bank. How much money does 
Pete have? 


3EST COPY AVAILABLE 
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Oops! Not Quite. 


Way to go! Your answer is correct! 
Unfortunately, Addy is not here. 
Try another location. 


as ts 


Figure 10 
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Appendix 


The Adventures of Addy Evaluation Form 


1. I understood the purpose of the game. 


2. I could do the math problems. 


3. I knew how to go from one place to another. 


4. I liked the pictures. 


5. [liked looking for Addy. 


6. I would play this game again. 


Comments: 
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